feat:Reduce unnecessary LLM calls from the respective channels #242

vaishcodescape · 2026-01-15T10:06:10Z

Closes #241

📝 Description

This PR fixes the unnecessary LLM API calls made by the user either in the Discord or GitHub channels by implementing a multi-layer caching and optimization system for message classification.

🔧 Changes Made

cache_helpers.py

Simplified hashing: Replaced with single XXHash128, reducing CPU overhead and reducing collision
Added MAX_MESSAGE_LENGTH (10KB): Truncates oversized messages before processing to prevent DoS attacks and excessive memory usage patches security issues
Bounded in-flight tracking: Replaced unbounded _inflight dict with TTLCache(maxsize=1000, ttl=120) to prevent memory leaks from orphaned requests
Fixed orphaned future bug: Added asyncio.CancelledError handling to prevent request hangs when tasks are cancelled

classification_router.py

Added LLM concurrency limiter: Implemented asyncio.Semaphore(10) to limit concurrent LLM API calls, preventing rate limit errors and cost explosions during traffic spikes
Added message length validation: Early return for messages exceeding MAX_MESSAGE_LENGTH with fallback classification

Cost Optimization Summary

Optimization	Impact
Pattern matching fast-path	~30-50% cached messages skip LLM entirely
Result caching	Repeated questions served instantly
In-flight deduplication	Concurrent identical requests share one LLM call
Concurrency limiter	Max 10 simultaneous LLM calls

📷 Screenshots or Visual Changes (if applicable)

N/A — Backend-only changes with no visual impact.

🤝 Collaboration

Collaborated with: N/A

✅ Checklist

I have read the contributing guidelines.
I have added tests that prove my fix is effective or that my feature works.
I have added necessary documentation (if applicable).
Any dependent changes have been merged and published in downstream modules.

Summary by CodeRabbit

Release Notes

New Features
- Complete redesign of the landing page with modern visual design, smooth animations, enhanced typography, and fully responsive layouts across all major sections for a significantly improved user experience.
- Optimized classification system with faster performance through improved request handling and intelligent pattern recognition.
Refactor
- Streamlined internal component export structure.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

…nce Tailwind CSS configuration for improved styling

…tton component

…ist sections with improved styling and functionality

…onents with QuickLink for improved hash navigation, and update Waitlist component for type safety

…default behavior in handleSmoothScroll function in Navbar

…avigation

…rrency control for message classification.

coderabbitai · 2026-01-15T10:06:21Z

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

This PR adds smart caching and pattern-based classification to reduce unnecessary LLM API calls, featuring TTL caches, concurrent request deduplication, and regex-based fast-path detection. Simultaneously, the landing page is redesigned using Material-UI and Framer Motion for improved visual consistency and interactivity.

Changes

Cohort / File(s)	Summary
LLM Caching & Optimization `backend/app/classification/cache_helpers.py`	New module introducing TTL-based in-memory caching, pattern-based fast classification with compiled regex patterns, async LLM call wrapper with in-flight deduplication, and message normalization. Exposes 9 public functions (make_key, key_for_normalized, get/set_cached_by_normalized, cached_llm_call, normalize_message, is_simple_message, cache_get/set) and module-level metrics counters.
Classification Router Integration `backend/app/classification/classification_router.py`	Integrates caching system into classification routing with module-level semaphore for concurrency control, message length validation, context-aware cache lookup, and deterministic fallback triage on LLM failures. Adds _fallback_triage method for error handling.
Frontend Export Cleanup `frontend/src/components/pages/index.ts`	Removes 4 re-exports: Dashboard, BotIntegrationPage, ContributorsPage, PullRequestsPage.
Landing Package Dependencies `landing/package.json`	Adds four Material-UI packages: `@emotion/react`, `@emotion/styled`, `@mui/icons-material`, `@mui/material`.
Landing Component Redesigns `landing/src/components/layout/Footer.tsx`, `landing/src/components/layout/Navbar.tsx`, `landing/src/components/sections/Hero.tsx`, `landing/src/components/sections/Waitlist.tsx`, `landing/src/components/ui/Button.tsx`	Comprehensive UI redesign: replaces Tailwind with MUI components and Framer Motion. Footer adds responsive grid layout, social links, back-to-top button, and animated transitions. Navbar integrates hash-based smooth scrolling and routing navigation. Hero refactors section layout with badge, animated gradient heading, dashboard preview card. Waitlist rebuilds form with MUI inputs, success state panel, and glassy aesthetics. New Button component with variant styling, icon slots, and hover/tap animations.
Landing Styling & Theme `landing/src/index.css`, `landing/tailwind.config.js`	Expanded design tokens (colors, spacing, shadows, radii, transitions), enriched typography scale, new utility classes for elevation/interactions/animations, new gradient/button/card styles, and extended Tailwind theme with primary/gray palettes, box-shadow presets, and animation aliases.

Sequence Diagram

sequenceDiagram
    participant Client as Message Source
    participant Router as ClassificationRouter
    participant Cache as Cache Layer
    participant Patterns as Pattern Matcher
    participant Dedup as In-Flight Dedup
    participant LLM as LLM Service

    Client->>Router: incoming message
    Router->>Router: validate message length
    Router->>Cache: normalize & lookup cache key
    Cache-->>Router: cache hit (if exists)
    Router-->>Client: return cached result

    rect rgba(200, 150, 255, 0.3)
        Note over Router,LLM: Cache miss path
        Router->>Patterns: check simple patterns
        Patterns-->>Router: match found?
        alt Pattern matched (fast-path)
            Router-->>Client: return pattern classification
        else No pattern match
            Router->>Dedup: check in-flight requests
            alt Request already in-flight
                Dedup-->>Router: wait for result future
                Dedup->>Cache: store result on completion
                Cache-->>Router: return stored result
            else First request for this message
                Router->>LLM: acquire semaphore & invoke
                LLM-->>Router: parse JSON response
                Router->>Cache: store result
                Router->>Dedup: fulfill all waiting futures
                Cache-->>Client: return cached result
            end
        end
    end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Professional aesthetic that's both modern and eye-catching #113: Modifies landing page UI components (Footer, Waitlist) for visual redesign, sharing file-level changes with this PR's frontend redesign.

Suggested reviewers

chandansgowda
smokeyScraper

Poem

🐰✨ Hops with glee at cache so bright,
Patterns match, no LLM in sight!
Landing page now gleams with MUI's grace,
Deduplication keeps the speedy pace! 🚀
Smart calls and smooth scrolls—what a sight!

✨ Finishing touches

📝 Generate docstrings

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aa66519 and 9ae96f4.

⛔ Files ignored due to path filters (2)

landing/package-lock.json is excluded by !**/package-lock.json
poetry.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

backend/app/classification/cache_helpers.py
backend/app/classification/classification_router.py
frontend/src/components/pages/index.ts
landing/package.json
landing/src/components/layout/Footer.tsx
landing/src/components/layout/Navbar.tsx
landing/src/components/sections/Hero.tsx
landing/src/components/sections/Waitlist.tsx
landing/src/components/ui/Button.tsx
landing/src/index.css
landing/tailwind.config.js

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

vaishcodescape added 8 commits January 11, 2026 00:16

feat:UI enhancement features added using material UI

74914b2

chore: update poetry.lock with new greenlet musllinux wheels and enha…

11f1e5a

…nce Tailwind CSS configuration for improved styling

fix: clean up imports in Navbar and improve button hover effect in Bu…

c589df7

…tton component

feat: integrate MUI icons and enhance footer, navbar, hero, and waitl…

364a351

…ist sections with improved styling and functionality

refactor: remove unused index.ts file, enhance Footer and Navbar comp…

f372653

…onents with QuickLink for improved hash navigation, and update Waitlist component for type safety

refactor: remove hover effect from ArrowUpIcon in Footer and prevent …

371a1e3

…default behavior in handleSmoothScroll function in Navbar

fix: add href attribute to MuiLink in Footer component for improved n…

af351a4

…avigation

feat: Implement LLM call caching, fast-path classification, and concu…

9ae96f4

…rrency control for message classification.

vaishcodescape closed this Jan 15, 2026

vaishcodescape reopened this Jan 15, 2026

vaishcodescape closed this Jan 15, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat:Reduce unnecessary LLM calls from the respective channels #242

feat:Reduce unnecessary LLM calls from the respective channels #242

Uh oh!

vaishcodescape commented Jan 15, 2026 •

edited

Loading

Uh oh!

coderabbitai bot commented Jan 15, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

feat:Reduce unnecessary LLM calls from the respective channels #242

feat:Reduce unnecessary LLM calls from the respective channels #242

Uh oh!

Conversation

vaishcodescape commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description

🔧 Changes Made

cache_helpers.py

classification_router.py

Cost Optimization Summary

📷 Screenshots or Visual Changes (if applicable)

🤝 Collaboration

✅ Checklist

Summary by CodeRabbit

Release Notes

Uh oh!

coderabbitai bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vaishcodescape commented Jan 15, 2026 •

edited

Loading

coderabbitai bot commented Jan 15, 2026 •

edited

Loading